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Novick  and  Lindley  (1978,  1979)  have  dealt  with  the  use  of 
utility  functions  for  applications  in  education  and  have  advocated 
the  use  of  the  standard  gamble  (von  Neumann  and  Morgenstern ,  1953) 
elicitation  procedure  with  the  addition  of  coherence  checking 
using  overspecification  and  a  least  squares  fit.  In  this  procedure 
utilities  are  inferred  from  probability  judgements  offered  by  the 
assessor  .  This  paper  describes  local  and  regional  coherence  pro¬ 
cedures  which  seek  utility  coherence  in  successive  restricted  domains 
of  the  parameter  space  as  preludes  to  overall  coherence  checking.  These 
procedures  and  some  others  are  viewed  as  possible  ways  of  avoiding 
anchoring  and  certainty  effect  biases  found  in  earlier  fixed 
probability  methods,  and  presumably  present  in  current  fixed  state 
procedures.  /  \j  r. 
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Int  roduct ion 

Karlier  approaches  to  utility  assessment  (Mosteller  and  Nogee, 
1951;  Sehlaifer,  1959,  1971;  Ralffa  and  Schlulfer,  1961;  Keeney  and 
Kail  fa,  1976;  and  so  on)  have  been  based  on  the  use  of  fixed  proba¬ 
bility  (FP)  assessment  procedures  in  which  utilities  are  elicited 
directly,  through  successive  bisections  of  the  parameter  space.  It  has 
been  suggested  (Mosteller  and  Nogee,  1951)  that  such  procedures  are 
easier  to  use  because  subjects  are  more  familiar  with  the  quantity  for 
which  the  utility  fuction  is  desired  than  thev  are  with  probabilities, 
which  they  are  required  to  state  in  the  standard  fixed  state  (SFS) 
procedure . 

Although  it  was  originally  thought  by  psychologists  that  utility 
theory  would  prove  useful  as  a  descript lve  model  (Swalm,  1966,  etc.), 
much  criticism  has  recently  been  levied  against  Its  use  in  that 
capacity.  As  principal  critics,  Kahneman  and  Tversky  (1978)  have 
proposed  an  alternative  descriptive  model.  The  main  basis  for  their 
criticism  is  that  the  phenomenon  described  by  Tversky  (1977)  as  the 
certainty  effect  results  in  preferences  that  violate  the  substitution 
axiom  or  expected  utility  hypothesis  of  utility  theory.  This  axiom 
(hypothesis)  states  that  preference  order  is  invariant  over  probability 
mixtures  and  is  formally  equivalent  to  the  assumption  that  there  is 
no  positive  or  negative  utility  for  the  act  of  gambling  Itself.  Speci¬ 
fically,  the  certainty  effect  is  the  phenomenon  that  the  utility  of  an 
outcome  seems  greater  when  it  is  certain  than  when  it  is  uncertain. 

This  effect  can  be  observed  when  subjects  are  presented  with  a  choice 
between  a  for-sure  and  a  chance  option,  the  choice  appearing  in  the 
standard  gamble,  regional  coherence,  and  local  coherence  assessment 
procedures  to  be  described  in  this  paper. 
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Utility  theory  as  considered  here  1b  used  as  a  normative 
model  rather  than  as  a  descriptive  model;  however,  it  is  still 
Important  to  consider  the  certainty  effect  because  Tversky  (1977)  has 
shown  that  even  when  subjects  were  told  that  their  preferences 
violated  utility  theory,  they  were  not  inclined  to  change  them 
(see  also  Kahneman  and  Tversky,  1972).  This  brings  into  question 
the  reliability  (coherence)  and  bias-free  character  of  utility 
assessment  procedures  obtained  through  both  fixed  state  and  fixed 
probability  methods  and  the  value  of  those  procedures  in  helping 
decision  makers  be  more  coherent.  However,  it  should  be  pointed 
out  that  the  gambles  studied  by  Kahneman  and  Tversky  and  those 
studied  by  Novick  and  Lindley  were  somewhat  different  and  that 
the  latter  authors  also  provided  incoherence  resolution  procedures. 

In  another  paper .Tversky  and  Kahneman  (1974)  described  several 
heuristics  used  by  persons  in  assessing  probabilities  and  the 
biases  to  which  they  could  lead.  Of  particular  interest  is  the 
anchoring  and  adjustment  heuristic,  which  Spetzler  and  Stael  von 
Holstein  (1975)  have  shown  can  reduce  the  reliability  with  which  the 
bisection  technique  used  by  fixed  probability  models  elicits  utilities 
This  heuristic  is  the  phenomenon  whereby  the  most  readily  available 
piece  of  information  often  forms  an  initial  basis  for  formulating 
responses  from  which  subsequent  responses  are  then  adjusted.  Since 
adjustments  from  this  basis  are  often  Insufficient,  a  central  bias 
results.  According  to  Slovic  (1972),  the  anchoring  and  adjustment 
heuristic  is  a  natural  strategy  for  easing  the  strain  of  integrating 
information.  The  anchor  serves  as  a  register  in  which  one  stores  fJ 
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impressions  or  the  results  of  earlier  calculations.  Why  adjustments 
from  the  anchor  are  usually  insufficient,  though,  is  unclear.  Slovic 
advances  two  hypotheses  to  explain  the  Insufficient  adjustment.  First, 
people  may  stop  adjusting  too  soon  because  they  tire  of  the  mental 
effort  involved  in  adjusting.  Alternatively,  the  anchor  may  take  on 
a  special  salience,  thus  causing  people  to  feel  that  there  is  less 
risk  in  making  estimates  close  to  it  than  in  making  estimates  that 
deviate  far  from  it.  According  to  Spetzler  and  Stael  von  Holstein 
(1975),  experimentation  has  shown  that  subjects  tend  to  produce  a 
central  bias  when, in  the  fixed  probability  bisection  method,  they  are 
asked  first  for  the  median  for  an  uncertain  quantity  and  then  for  the 
quartl les. 

i.ater,  in  reviewing  the  role  of  nan-nachlne  systems  in  decision 
analysis,  Slovic,  Kischhoff,  and  Lichtenstein  (1977)  suggested  that 
human  factors  such  as  the  ways  in  which  variations  in  instructions 
or  inf ormat ional  displays  affect  people' i  performance  are  Important 
and  should  be  studied  in  more  detail.  Questions  of  complexity  and 
representativeness  of  material  seem  to  have  substantial  effect  on 
assessors  responses  (Fischhoff,  Slovic,  and  Lichtenstein,  1977;  Vlek, 
1973).  The  study  of  such  factors  might  lead  to  an  assessment  procedure 
that  minimizes  the  judgmental  basis  and  heuristics  dt'scribed  earlier. 
This  position  was  strengthened  by  the  discussion  of  Fischhoff,  Slovic, 
and  Lichtenstein  (1979).  A  consideration  of  these  ideas  promoted  the 
development  of  a  new  format  introduced  later  in  this  paper. 
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Kxtenstve  previous  work  in  this  area  has  raised  more  questions 
concerning  bias  and  coherence  than  it  has  provided  answers.  An  ap¬ 
parently  pessimistic  mood  prevails,  not  inappropriately,  given  the 
Importance  of  the  questions  that  have  been  raised  (Hogarth,  1975; 

Slovic,  1975;  and  Fischhoff,  Slovic  and  Lieht enst efn ,  1979),  Nevertheless, 
the  very  extensiveness  of  this  research  must  itself  imply  a  high  assess¬ 
ment  for  the  product  of  the  probability  of  resolving  these  difficulties 
and  the  value  of  this  outcome.  The  position  taken  here  is  that  bias  and 
incoherence  can  be  reduced  if  (l)  elicitations  are  carefully  fashioned 
in  a  Computer-Assisted  Data  Analysis  (CADAl  environment  (N  ick,  Woodworth, 
Hamer,  Chen,  Chuang,  Libby,  Molenaar,  Lewis  &  Isaacs,  1980),  (2) 
assessors  are  aided  in  resolving  incoherence,  and  (3)  if  the  assessments 
concern  states  and  actions  that  are  meaningful  and  important  to  the 
assessor  at  the  time  the  assessment  is  made. 

Consider  a  variable  9  and  the  utility  function  U(9)  for  which 
assessment  is  required.  In  most  applications  9  will  be  a  real  vari¬ 
able,  such  as  grade  point  average  (CPA) ,  but  this  is  not  necessary. 

Although  the  contrary  assumption  is  sometimes  made,  it  seems  sensible 
to  us  to  demand  that  a  utility  function  he  bounded. 

There  are  two  standard  approaches  to  assessing  a  utility  function: 
fixed  probability  and  fixed  state.  In  the  former,  the  subject  is  pre¬ 
sented  with  a  gamble  on  two  values,  or  states,  9^  and  9,  with  a  fixed 
probab ill ty-p ,  sav,  for  9^  and  1  -  p  for  9,  -  and  is  required  to  choose 
an  Intermediate  state  9  ^  such  that  he/she  is  indifferent  with  respect 
to  the  gamble  and  9(  for  sure.  In  applications,  typically  p  -  1/2 
because  this  gamble  ts  easiest  for  assessors  (subjects^  to  understand. 
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In  the  fixed-state  method,  the  states  0  ,  O^,  and  0  ^  are  fixed, 

0^  still  being  intermediate  between  9^  and  6^.  The  subject  is  required 
to  state  a  probability,  p,  such  that  he/she  is  indifferent  between  6^ 
for  sure  and  the  following  gamble:  0^  with  probability  p  and  0^  with 
probability  1-p.  If  0^  and  0^  have  utilities  of  1  and  0,  respectively, 
the  gamble  has  expected  utility  p,  the  indifference  probability  assigned 
to  0^. 


In  the  fixed-state  method,  let  us  suppose  that  a  number  of  states, 

9_,  9  .  •••.  0,,  ,,  are  selected.  We  shall  further  suppose  that  these 
0  1  N+l 

states  are  ordered  in  the  sense  that  0  is  preferred  to  0^  whenever 
j  >  i;  in  particular,  0^+^  is  the  best  and  0^  the  worst  state.  Then 
the  utility  function  U(0)  will  be  strictly  increasing. 

Without  loss  of  generality,  the  utility  for  0N+|  can  be  assigned 
the  value  1  and  that  for  9^  can  be  assigned  the  value  0,  thus  placing 
bounds  on  the  utility  values  to  be  assigned  to  the  various  states.  We 
must  then  find  N  such  values:  U(O^),  U(02),  •••.  U  (0^).  We  first  con¬ 
sider  adjacent  gambles,  that  is,  a  situation  in  which  the  subject  is 
asked  to  compare  the  sure  outcome  0  (l<_n^_N)  against  a  gamble  with  pos¬ 
sible  outcomes  9n_1  and  ®n+i>  representing,  because  of  the  ordering  of 
the  states,  situations  respectively  worse  and  better  than  0^.  Speci¬ 
fically,  after  a  brief  review  of  the  meaning  of  probability,  the  subject 
is  asked  to  state  the  probability  p^  for  and  consequently  1  -  pn 

for  0  ,,  that  makes  him/her  indifferent  with  respect  to  the  gamble 

n-1 


and  0  for  sure.  Writing  U(0  )  “  u  (so  that  uf 


0, 


n  -  n'  n  '  '0  w*  Vl 

equating  the  expected  utilities  for  the  two  situations  gives  us 


1)  and 


u 


p  u  ...  +  (1  -  P  )u  . 

n  n+l  n  n-1 


: 


-7- 


11  this  Is  done  for  all  n,  I'ti-N,  we  have  N  equations  In  N  unknowns 
and  aside  from  exceptional  cases,  the  utilities  are  uniquely  determined. 
The  solution  ts 

u  .  -  c  /C 
n+ 1  n  N 

for  Omv-N,  where 

n  n 

G  -  i:  F.,  F  -  11  f.,  and  f  -  (1  -  p  )/p  . 

n  1-0  1  n  1-0  1  1 

Suppose  a  subject  has  responded  to  the  N  question  previously 

considered  and,  from  the  answers  given,  his/her  utilities  u^,  u.,, 

....  have  been  determined.  Suppose  also  that  he/she  is  asked  to 

consider  a  gamble  that  will  vleld  either  t'n+,  or  0  2«  against  0R 

for  sure.  Then  the  probability  q  ,  associated  with  0  ,,  satisfies 

'  hi  n*2 

u  -  q  u  +  (1-q  )u  .. 

n  n  n+2  n  n-2 

For  the  fixed  state  standard  gambles  procedure  the  suggestion 
offered  by  Novick  and  l.indlev  Is  that  to  exploit  coherence  fully, 
we  must  ask  for  more  probability  assessments  than  are  needed  to  cal¬ 
culate  the  utilities  and  then  compare  them  for  coherence.  The 
Idea  of  requiring  the  experimenter  to  give  mote  than  the  minimum 
number  of  Judgments  in  titting  a  personal  probability  distribution 
has  been  used  hv  Pratt,  Kalita,  and  Schlaifer  (19b5)  aid  has  been 
exploited  systematically  both  for  the  assessment  of  probabilities 
.uid  utilities  in  the  development  of  the  Computer  Assisted  Data  Analysis 
Monitor  (CADA)  iNovick,  197.1,  197S).  In  the  context  of  utility  assess¬ 
ments,  the  Idea  has  been  used  hv  Becker,  DeGroot,  and  Marschak  (19t>3) 
with  fixed  probability  assessments,  and  we  shall  discuss  this  presently. 
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Experience  shows  us  that  assessors  are  almost  always  incoherent 
but  readily  attempt  to  resolve  their  incoherences  when  these  are 
brought  to  their  attention  (cf.  MacCrimmon,  1965).  It  may,  however, 
be  true  that  one  kind  of  gamble  (e.g.,  adjacent  gambles)  may  introduce 
one  kind  of  systematic  bias  and  another  kind  (e.g.,  extreme  gambles) 
may  introduce  a  second  kind  of  bias.  Therefore,  rather  than  just 
asking  the  subject  to  revise  some  of  his/her  assessments,  Novick  and 
Lindley  (1979)  suggest  assisting  the  subject  by  providing  a  least 
squares  fit  in  the  log-odds  metric  for  the  N  undetermined  utility 
values.  A  computer  program  has  been  written  to  carry  out  the  inter¬ 
rogation  of  the  assessor  and  to  perform  the  least  squares  fit  and 
is  available  on  the  CADA  Monitor  (Novick  1973,  1975;  Novick,  Isaacs, 
and  Dekeyrel,  1977;  Isaacs  and  Novick,  1978). 

In  any  comparison  of  fixed  state  with  fixed  probability  assess¬ 
ments,  the  role  of  coherence  seems  to  us  to  play  a  dominant  role. 
Although  subjects  often  prefer  fixed  probability  assessments,  especi¬ 
ally  when  the  probability  is  1/2,  exploiting  coherence  in  this  context 
is  harder  than  with  the  fixed  state  procedure.  For  example,  suppose, 
as  usual,  that  a  subject  is  asked  for  the  certainty  equivalent  of  a 
gamble,  at  even  odds,  on  the  best  (6N+^)  and  worst  (9q)  states.  Let 

his/her  stated  value  be  8  ,  say,  having  u(0  )  *  1/2.  The  subject  is 

m  m 

then  asked  for  the  certainty  equivalents  for  even-odds  gambles  on 

(0  ,0  )  and  (0  ,  0  , ) .  If  these  values  are  denoted  by  0  and  8_  , 

0  in  in  IttI  1  2. 

respectively,  then  the  utilities  of  0^  and  are  u(0^)  »  1/4  and 

u(0o)  =  3/4.  Finally,  he/she  is  asked  to  consider  an  even-odds  gamble 

on  0  and  But  it  is  rather  transparent  that  for  coherence  the 

result  must  be  0  ,  so  that  the  four  judgments  can  scarcely  be  consi- 
m 

dered  independent.  In  this  field  (as  in  other 
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mcaaurement  fields)  obtaining  Independent  repetitions  of  the  same 
assessment  is  hardly  ever  possible,  thus  the  emphasis  ought  to  be  on 
Independent  assessments  of  related  quantities.  This,  we  feel,  is  more 
nearly  achieved  with  the  fixed-state  assessments.  The  above  discussion 
Is  taken  with  some  condensation  from  Novick  and  Llndley  (1979). 

The  question  that  must  now  be  addressed  Is  whether  the  incoherence 
resolution  of  the  least  squares  method  described  in  SFS  above  avoids  the 
certainty  and  anchoring  effects  or  whether  better  methods  can  be  found, 
rhe  remainder  of  this  paper  will  be  devoted  to  describing  a  refinement 
in  the  least  squares  SFS  procedure  and  in  describing  three  new  proce¬ 
dures  that  more  directly  address  these  biasing  effects. 

A  word  concerning  ease  of  response  may  be  in  order.  Mosteller 
was  certainly  correct  in  saying  that  FT  is  easier  than  FS ,  and,  indeed, 
without  interactive  conversational  computing  facilities  an  FS  assessment 
procedure  may  well  be  unbearably  difficult.  With  conversational  computing, 
U'wever,  an  FS  procedure  is  bearable  and  there  is  no  reason  to  believe 
the  the  easier  method  is  more  bias  free.  Indeed,  the  contrary  could 
be  true. 

In  the  current  version  of  the  SFS  procedure  on  CADA,  subjects  are 
given  situations  consisting  of  a  for-sure  and  a  chance  option  on  grade 
point  averages  in  the  range  0-4  and  are  asked  for  the  probabilities  that 
make  them  indifferent  with  respect  to  the  two  options  in  each  situation 
(i.e.,  their  Indifference  probabilities).  The  indifference  probabilities 
for  the  fixed  state  gambles  are  elicited  using  one  of  two  formats  for 
presenting  the  gambles.  Format  two  requests  a  direct  magnitude  esti¬ 
mation  as  illustrated  earlier.  Format  one  asks  for  preferences  for 
gambles  or  sure  things  for  p  values  .1,  .9,  .2,  .8,  etc.  or  .9,  .1,  .8, 

.2  etc.  with  jeroing  in  on  the  Indifference  point. 


FORMAT  ONE 


3.0  p  chance  indif f erent=0 

2.5  for  sure  for  sure  =1 

2.0  1-p  chance  chance  c2 

restart  "3 

Which  would  you  prefer  if  p  were. XX  ? _ (This 

question  was  repeated  using  the  following  p  values-- 
.1,  .9,  .2,  .8,  ... — until  p  was  found  to  be  between  .5 
and  .6,  say.  Then  the  questioning  procedure  used  p  values 
of  .52,  .58,  .59,  ...  until  the  subject's  indifference  p 
had  been  determined.) 


FORMAT  TWO 


for  gamble 

sure  with  prob  p  with  prob  1-p 

2.5  3.0  2.0 


p  that  makes 
you  indifferent 
? 


Table  1.  Formats  for  the  SFS  utility  assessment  procedure 
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Format  two,  tuc  direct  probability  assessment  format  is  the  one 
used  by  Novick  and  Lindley,  (1979).  Format  one,  the  end a- in 
procedure, has  been  advanced  as  a  method  for  avoiding  anchoring.  Since 
Indifference  points  are  typically  between  .2  and  .8  any  initial  anchor 
(.1  or  .9)  is  erased  before  any  careful  judgment  must  be  made.  Also, 
the  starting  values  alternate  between  .1  and  .9  thus  avoiding  any 
constant  ordering  effect.  It  is  our  as-yet-unsubstant iated  belief 
that  this  format  is  both  easier  and  less  subject  to  anchoring  than 
format  two.  This  format  is  now  used  with  several  assessment  procedures. 

In  order  to  avoid  the  documented  biases  of  the  certainty 
effect  in  utility  assessment,  a  new  procedure  has  been  considered: 
the  paired  binary  gambles  (PBC.)  procedure.  This  procedure  is 
illustrated  in  Table  2  below.  The  paired  gambles  in  the  table  can  be 
abbreviated  as  (1.5  3.0,  2.0  2.5). 


It  has  previously  been  used  in  a  f  ixed  probability  paradigm  (Kneppreth, 
Gustafson,  l.eifer,  &  Johnson,  1974).  Suppes  and  Walsh  (1959)  have 


considered  such  gambles  strictly  in  the  sense  of  determining  preferences 
between  the  two  situations,  without  eliciting  either  indifference 
probabilities  or  equivn lance  points. 

The  obvious  hope  is  that  the  PBC  procedure  will  avoid  the  certainty 
effect  because  the  comparison  is  between  two  sets  of  gambles,  and  thus 
does  not  involve  the  for-sure  option.  We  have  used  PBG  in  some  informal 
assessments  but  have  not  yet  been  convinced  of  its  usefulness.  First, 
it  is  difficult  even  for  experienced  subjects.  Fatigue  and  boredom 
are  definite  problems.  We  are  not  sure  that  there  is  not  a  bias  in 
that  one  situat  ion  always  compares  two  adjacent  states  while  the 
other  always  describes  two  states  twice  removed.  V.'e  have  not  discarded 
this  procedure,  but  we  feel  that  retlnements  may  be  necessary  if  it 
is  to  be  useful. 

Next  we  define  the  Regional  Coherence  (RC  procedure).  In  the 
RC  procedure.  Indifference  probabilities  are  elicited  separately  for 
two  SFS  gambles  using  the  ends- in  format.  Subjects  are  then  presented 
with  a  table  allowing  the  initial  gambles  (situations  1  and  2  with 
their  indifference  probabilities)  and  two  additional  gambles 
(situations  3  and  4).  They  are  told  that  their  initial  responses  imply 
certain  specific  indifference  probabilities  for  the  two  new  gambles. 
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Table  l  illustrates  the  latter  part  of  this  procedure. 


S i tuat  ions 

i 

_ _ _  1  _  _ _ 

—  3-  — 

- 4  ___ 

p  chance 

1.50 

2.00 

2.00 

2.00 

for  sure 

l  .00 

1.50 

1 . 00 

1.50 

l  —  p  chance 

0.  50 

1.00 

0.50 

0.50 

p- .  53 

P-.58 

O 

1 

C- 

P-.75 

Table  3.  RC  Procedure 


Assessors  .mo  then  given  the  opportunity  to  change  the  Indiffer¬ 
ence  probabilities,  two  at  a  time,  until  they  are  Indifferent  In 
all  four  situations.  They  choose  the  two  situations  for  which  they 
wish  t»  change  the  indifference  probabilities  and  the  nagnitude 
estimation  format  is  then  implemented  to  generate  the  revised 
probab  1 1 It  ies. 

The  final  procedure  is  called  the  local  coherence  (LC)  procedure. 
This  procedure  presents  subjects  with  two  types  of  hypothetical 
choice  situations:  (1)  a  for-sure  and  a  chance  option  (the  standard 
gamble)  and  (2)  two  chance  options.  The  ends-in  format  is  used  to 
elicit  an  Indifference  probability  for  the  first  situation,  after  which 
the  subject  is  told  that  that  response  implies  that  he/she  should  be 
indifferent  with  respect  to  the  two  options  in  situation  2.  Note  that 
t  lie  subject  only  specifies  the  Indifference  probability  for  the 
standard  gamble.  Table  4  below  Illustrates  this  procedure.  The 
probabilities  for  the  second  situation  are  uniquely  determined  by  that 


spec i i lent  Ion . 


1 A 


SITUATION  l 

SITUATION 

2 

A.  00 

.75  chance 

option  one 

.19  A. 00 

opt  ion  two 

3.00 

for  sure 

3.00 

.25 

1.00 

.25  chance 

.81  1.00 

.75 

Table  A.  l,C  Procedure 

If  the  subject  is  not  indifferent  in  both  situations,  he/she  modifies 
the  situation  1  indifference  probability  and  then  is  again  presented  with 
a  table  similar  to  Table  A  above.  This  continues  until  the  subject  is 
indifferent  with  respect  to  the  two  situations. 

In  choosing  a  fixed  state  assessment  procedure  we  are  free 
to  select 

(1)  A  response  format 

a.  ends-in 

b.  direct  specification 

(2)  A  comparison  format 

a.  standard  fixed  state 

b.  paired  binary  gambles 

c.  regional  coherence 

d.  local  coherence 

(3)  Overall  coherence  checking  bv  least  squares 

a.  yes 

b.  no 


The  temptation  for  a  person  trained  in  both  psychology  and  statistics 
to  undertake  the  experimental  comparison  using  some  subset  of  a  ?  by  A 
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by  2  factorial  design  is  overpowering.  Indeed  the  tooling-up  for  this 
experiment  has  begun  including  a  further  ccxnparison  with  the  fixed  proba¬ 
bility  method  and  an  Investigation  of  comparative  bias  for  central 
and  extreme  values  of  0.  For  a  Bayesian  statistician,  yielding  to  this 
temptation  leads  to  a  compulsion  to  state  a  prior  distribution.  In  the 
absence  of  a  precisely  stated  model  this  is  not  possible,  but  it  is 
possible  to  state  some  general  beliefs.  I  shall  now  do  this  and  also 
invite  you  to  attend  the  Psychometric  Society  meetings  in  May  of  1980 
where  I  shall  report  on  the  results  of  these  experiments. 

First,  I  believe  that  SFS  with  overall  LSQ  coherence  checking  will 
prove  to  be  good  but  not  best.  Subjects  find  it  hard  to  make  unaided 
adjustments.  As  a  result,  Incoherence  will  remain  high,  but  overall 
fits  will  be  tolerably  good  (p».7).  However,  we  are  working  on  improve¬ 
ments  that  could  make  this  procedure  more  attractive.  I  believe  that 
the  ends-in  format  will  be  preferred  over  direct  magnitude  estimation 
and  will  reduce  the  anchoring  effect  (p-.8).  This  may  not  hold  for 
very  experienced  assessors  who  may  find  it  tedious. 

I  believe  that  PBG  will  be  unpopular  and  ineffective  unless  we  find 
some  simplification  ( p=- . 9 )  .  At  present  it  is  difficult  and  fatigueing 
and  responses  tend  to  be  less  than  carefully  considered. 

I  believe  that  regional  and  local  coherence  will  both  be  useful 
and  both  will  largely  eliminate  anchoring  and  adjustment  biases  (p-.8). 

I  believe  the  regional  coherence  will  be  preferred  by  inexperienced 
users  (p’.6)  and  local  coherence  by  experienced  users  (p“.7,  but  per¬ 


haps  only  professional  statisticians).  Local  Coherence  provides  a  display 
of  the  large  effect  on  extreme  comparisons  of  minor  adjustments  in  non- 


extreme  comparisons.  It  is  a  powerful  tool  for  locating  the  most  de¬ 
sirable  point  in  the  probability  range  p  +  .025.  It  is  unclear  to  me 
whether  overall  least  squares  overfitting  will  be  useful  in  conjunction 
with  the  RC  or  LC  procedures,  but  my  prior  probability  is  .6  that  it  is. 

Finally,  for  certain  points  I  have  high  personal  probability. 
Utility  elicitation  procedures  can  currently  be  conducted  with  accuracy 
and  ease  on  CADA.  Indeed,  they  can  be  conducted  with  sufficient  ease 


and  potentially  with  sufficient  freedom  from  bias  as  to  make  applications 
of  utility  theory  to  education  entirely  feasible  when  the  assessor  is 
confronted  with  a  specific  problem  of  interest  and  importance,  and  when 


that  problem  is  presented  clearly  and  unambiguously. 
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5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasmor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22  33 
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Army 


Air  Force 


Director,  Training  Develop  lent 
U.S.  Army  Administration  Center 
ATTN:  Dr.  Sherrill 
Ft.  EWijnnin  Harrison,  IN  46218 

Dr.  Frederick  Steinheiser 
U.  f.  Army  F"Si-rch  Institute 
5r,r' '  Eisenhower  Avenue 
Alexandria,  VA  ??37  j 

Dr  .  Joseph  hard 
IKS.  Army  Reserrch  Institute 
5^01  Eisenhower  Avenue 
Alexandria  ,  VA  22  :33 


1  Air  Force  Human  Resources  Lab 
AFBRL/PED 

Brooks  AFP,  IX  78? 35 

1  Air  University  Library 
AUL/LSE  76/443 
Maxwell  A.FE,  AL  36 1 1  ? 

1  Dr  .  Pin  l  i  p  De  Leo 
AFHRL/TT 

Lowry  AFD ,  CC  802^0 

I  DR.  G.  A.  ECKSTKAMD 
AFHRL/AS 

WRIGHT-PATTEPSOM  AFD,  OH  46433 

1  Dr.  Genevieve  Haddad 
Program  1  ^nager 
Life  Sciences  Directorate 
AFC  SR 

Bolling  AFD,  DC  20j32 

1  CDIK  MERCER 

Cr.'ET  LIAISON  OFFICER 
AFHRL/FLYING  TRAINING  DIV. 
WILLIAilS  AFB,  AZ  85224 

1  Dr.  Ross  L.  Morgan  ( AFHRL/ASR) 
Wright  -Patterson  AFB 
Ohio  454J3 

1  Dr.  Roger  Pennell 
AFHRL/TT 

Lowry  AFB,  CO  80230 

1  Personnel  Analysis  Division 
HQ  USAF/DPXXA 
Washington,  DC  20330 

1  Research  Branch 
AFMPC/DPMYP 

Randolph  AFB,  TX  76148 

1  Dr.  Malcolm  Ree 
AFHRL/PED 

Brooks  AFB,  TX  78235 
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Air  Force  Marines 


1  Dr.  Marty  Rockway  (AFHRL/TT)  1 

Lowry  AFB 

Colorado  3023D 

1  Jack  A.  Thorpe,  Capt,  USAF 

Program  Manager  1 

Life  Sciences  Directorate 

AFOSR 

Bolling  AFB,  DC  20332 

1  Brian  K.  Waters,  LCOL,  USAF  1 

Air  University 
Maxwell  AFB 
Montgomery,  AL  3C11? 


H.  William  Greenup 
Education  Advisor  (E031) 

Education  Center,  MCDEC 
Quantico,  VA  22131* 

Director,  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg .  2009 
Quantico,  VA  2213^ 

DK.  A. L.  SLAFKOSKY 
SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 

HC,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20J80 
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CoastGu  ir  1 


1  Mr.  Richard  La n ter. -nan 

PSYCHOLOGICAL  RESEARCH  (G-F 
tl.S.  CCA  ST  GUARD  HO 
WASHINGTON,  PC  20V)0 

I  Pr  .  Thomas  Warm 

U.  G.  Coast  Guard  Institute 
P.  0.  Substation  IS 
Oklahoma  City,  OK  ’JIM 


Other  IV) P 


-1/62) 


12  Defense  Docunenta t ion  Center 
Cameron  Station,  LUdg.  S 
Alexandria ,  VA  22.114 
Attn:  TC 

1  Dr.  rV'Xter  Fletcher 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILGON  LLVP . 

ARLINGTON,  VA  22209 

1  Dr.  Willi  am  irah.jir. 

Test  tng  Director  il< 

HEPCO'1 

Ft .  Slier  id. ill ,  IL  600  <7 

1  Military  Assistant  l'or  Training  ..nd 
Pe  r  so  t  m  c l  Technology 
Office  of  the  Under  Secretary  of  Pe f 
for  Research  A  Engineering 
Room  )D1'(’,  Hie  Pentagon 
W  .  sh  l  ng  ton  ,  IX.-  20  }0  1 

1  MAJOR  Wayne  Sellmun,  USAF 

Office  of  the  Assistant  Secretary 
of  Defense  (  MKAAL ) 

3H930  Tiic  Pent  agon 
Washington.  DC  20301 
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Civil  Govt 


Civil  Govt 


1  Dr.  Susan  Chipmcn 

P-iSic  Skills  Prog  ran 
National  Institute  ol'  Education 
1200  IQ th  Street  Nil 
Washington,  IX)  2020? 

1  Dr.  Jill iam  Gorham,  Director 
Personnel  RAD  Center 
Office  of  Personnel  fannginent 
non  £  street  I!  J 
'..'ashing ton  ,  DC  20*415 

1  Dr.  Joseph  I.  Lipson 

Division  of  Science  Education 
Room  W-C.I? 

National  Science  foundation 
Washington,  DC  20550 

1  Dr.  John  Hays 

National  Institute  of  Education 
1200  19th  Street  NW 
!!  'Shington  ,  DC  2020'" 

1  Dr.  Arthur  Helmed 

National  Intitute  of  Education 
1200  19th  Street  MW 
Washington,  DC  2020S 

1  Dr.  Andrew  R.  Molnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Lalitha  P.  Srinathanan 

Environmental  Impact  Studies  Division 
Argonne  National  Laboratory 
9700  S.  Cass  Avenue 
Argonne,  IL  bO*J39 

1  Dr.  Jeffrey  Schiller 

National  Institute  of  Education 
1200  19th  St.  NW 
Washington,  DC  20203 


1  Dr.  Thomas  G.  Sticht 
Easio  Skills  Program 
National  Institute  of  Education 
1200  l'jth  Street  NW 
Washington,  DC  20203 

1  Dr.  Vern  W.  Urry 

Personnel  RAD  Center 
Office  of  Personnel  Hanagraent 
1900  E  Street  NW 
Washington,  DC  20*415 

1  Dr.  Frank  Withrow 

U.  2.  Office  of  Education 
400  Gth  Street  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  A  Cognitive  processes 
National  Science  Foundation 
Washington,  DC  20550 
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Non  Govt 


Non  Govt 


1  Dr.  Earl  A.  Alluisi 
HQ,  AFHRL  ( AF3C) 

Brooks  AFH.  TX  78235 

1  Dr.  Erling  D.  Anderson 
University  of  Copenhagen 
Studiestraedt 
Co  penhugen 
DENMARK 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Alan  Baddeley 

Medical  Research  Council 

Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CP2  2EF 
ENGLAND 

1  Dr.  Isaac  Be  jar 

Educational  Testing  Service 
Princeton,  NJ  03450 

1  Dr.  Warner  Birice 
Streitkraefteamt 
Rosenberg  5j00 
Bonn,  West  Germany  D-5300 


1  Dr.  R.  Darrel  Bock 

Department  of  Education 
University  of  Chicago 
Cnicngo,  IL  60637 


Dr.  Nicliolas  A.  Bond 
Dept,  of  Psychology 
Sacramento  State  College 
600  Jay  Street 
Sacramento,  CA  95319 

Dr.  David  G.  Bowers 
Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  HI  43106 


1  Dr .  Robert  Erennan 

American  College  Testing  Programs 

P.  O.  Eox  163 

Iowa  City,  IA  52240 

1  DR.  C.  VICTOR  BUNDERS0N 
WICAT  INC. 

UNIVERSITY  PLAZA,  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  84057 

1  Dr.  John  E.  Carroll 
Psychometric  Lab 
Univ.  of  No.  Carolina 
Davie  Hall  0 1 3A 
Chapel  Hill,  NC  27514 

1  Charles  Myers  Library 
Livingstone  House 
Livingstone  Road 
Stratford 
London  E15  2LJ 
ENGLAND 

1  Dr.  Jonn  Chiorini 
Li t to n-Mel Ionics 
Box  1236 

Springfield,  VA  22151 

1  Dr.  Kenneth  E.  Clark 

College  of  Arts  &  Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

1  Dr.  Norman  Cliff 
Dept,  of  Psychology 
Univ.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  William  Coffman 
Iowa  Testing  Programs 
University  of  Iowa 
Iowa  City,  IA  52242 
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Mon  Govt 


f.'on  Govt 


1  Dr.  Meredith  Crawford 

Department  of  Engineering  Administration 
George  Washington  University 
Suite  305 

2101  L  Street  N.  V.'. 

Washington ,  DC  20037 

1  Dr.  Hans  Cronbag 

Education  Research  Center 
University  of  Leyden 
Hoerhaavelaan  2 
Leyden 

The  NETHERLANDS 

1  Dr  .  Emmanuel  Donchin 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  MAJOR  I.  M.  EV0NIC 

CANADIAN  FORCES  PERS.  APPLIED  RESEARCH 
1107  AVENUE  ROAD 
TORONTO,  ONTARIO,  CANADA 

1  Dr.  Leonard  Feldt 

Lindquist  Center  for  Neasurment 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Richard  L.  Ferguson 

The  Am  eric  .an  College  Testing  Program 

P.0.  Pox  168 

Iowa  City,  IA  52240 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Dr.  Gerhardt  Fischer 
Liebig a sse  5 
Vienna  1010 
Austria 

1  Dr.  Donald  Fitzgerald 

University  of  Mew  England 
Armidale,  Mew  South  Wales  2351 
AUSTRALIA 


1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 

1  Dr.  John  R.  Frederiksen 
Lola  Beranek  h  Neuman 
50  Moulton  Street 
Cambridge,  FA  02138 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
2939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Ross  Greene 
CTB/KcGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  939*10 

1  Dr.  Alan  Gross 

Center  for  Advanced  Study  in  Education 
City  University  of  New  York 
New  York,  NY  10036 

1  Dr .  Ron  Hambleton 
School  of  Education 
University  of  Massechusetts 
Amherst,  MA  01002 

1  Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93 1 06 

1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Library 

HumRRO/Western  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 
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1  Dr.  Steven  Hunk  i 

Deportment  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 

1  Dr.  Fir  1  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  9? 1  PS 

1  Dr  .  Huynh  Huynh 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  2920° 

1  Dr.  Cirl  ,1.  Jensen.) 

Call, ud  e  t  College 
Kendall  Green 
Wash  in,’ ton  ,  DC  2000'’ 

1  Dr.  Arnold  F.  Kanarick 
Honeywell,  Inc. 

2600  Ridgeway  Pkwy 
Minneapolis,  M.H  59 'll  3 

1  Dr.  John  A.  Kents 

University  of  Newcastle 
Newcastle,  flow  South  Wales 
AUSTRALIA 

1  Mr.  Marlin  Kroger 
1117  Via  Go 1 eta 

Palos  Verdes  Estates,  CA  9927*1 

1  LCOL.  C.R.J.  LAFLEUR 
PERSONNEL  APPLIED  RESEARCH 
NATIONAL  DEFENSE  HQS 
101  COLONEL  15 Y  DRIVE 
OTTAWA,  CANADA  K 1 A  OK 2 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
University  of  Illinois 
Champaign,  TL  61820 
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Non  Govt 


1  Faeulti  it  Sociale  Vie  ten  so  happen 
Ri jksuniversit eit  Groningen 
(Dude  Boter  ingrstra.it 
Groningen 
NETHERLANDS 

1  Dr.  Robert  Linn 

College  of  Education 
Un  1  vi  r si  t  y  of  Ill  mo i s 
Urban.) .  !L  61801 

1  Dr.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  C^SUO 

1  Dr.  Gary  Marco 

Educational  Testing  Service 
Princeton,  NJ  0°*<50 

1  Dr.  Scott  Maxwell 

Department  of  Psyclxilogy 
University  of  Houston 
Houston,  TX  77025 

1  Dr.  Sam  Mayo 

Lo yo  1  a  Un i v cr s i t y  of  Ch i e ago 
C1i leago ,  IL  60691 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0,  box  751 
Portland ,  OR  97207 

1  MR.  LUIGI  PETRULL.0 

2*131  N.  EDGEWOOP  STREET 
ARLINGTON,  VA  22207 

1  DR.  STEVEN  M.  PINE 
*<950  Douglas  Avenue 
Golden  Valley,  MN  55**16 

1  DR.  DIANE  M.  RAMSEY-KLEE 

R-K  RESEARCH  A  SYSTEM  DESIGN 
39*<7  RIDGEMONT  DRIVE 
MALI  HU,  CA  90265 
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No  n  Go  v  t 


i  min.  r:-:t.  t:.  rauch 
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HUNDESM INISTERIUM  PER  VSRTEID1GUNG 
POoTFACM  161 
53  3011*1  1,  GERMANY 

1  Dr.  Peter  B,  Read 

Social  Science  Research  Council 
605  Third  Avenue 
'lew  York,  NY  10016 

1  Dr.  Mirk  P.  Reckase 

Educational  Psychology  Pept. 
University  of  Missouri -Col unbin 
l?  Mill  Hall 
Col  mb  l -I,  MO  65201 

I  Ur.  Andrew  M,  Rose 

American  Institutes  for  Research 
1065  Thomas  Jefferson  St.  NW 
Washington ,  PC  20007 

1  Dr.  Leonard  L.  Rosenbaum,  Qtain/ixn 
Department  of  Psychology 
Montgomery  College 
Rockv il lc  ,  MP  20350 

1  Dr.  Ernst  Z.  Rothkopf 
Pell  Laboratories 
500  Mountain  Avenue 
Murray  Hill,  MJ  0797*1 

1  Pr  .  Donald  Rubin 

Educational  Testing  Service 
Princeton,  NJ  0R1I50 

1  Dr.  l.3rry  Rudner 
Gallnudet  College 
Kendall  Green 
Washington,  DC  20002 

1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia ,  SC  29208 


1  PROF.  FUMIKO  5AMEJIMA 
DEPT.  (iF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916 

1  Pr  .  K. iz  io  Ghigemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kawjuchi ,  Sendai  932 
J  A  PA  N 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  «Uj05 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yule  University 
box  1 1 A ,  Yale  Station 
New  Haven,  CT  06520 

)  DR.  PATRICK  3UPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  9*1305 

1  Dr.  Hnriharan  Swaminathan 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  M.A  01003 

1  Dr.  Brad  Sympson 

Office  of  Data  Analysis  Research 
Educational  Testing  Service 
Princeton,  NJ  C35M 

1  Dr.  Kikumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  1L  61801 
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1  Pr .  Maurice  Tatsuoka 

Department  of  Educational  Psychology 
University  of  Illinois 
Champa ign ,  IL  61ND1 

1  Pr .  David  Thisaen 

Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  fSOUR 

1  Hr  .  Robert.  Tsutr.k.awa 
Dept .  of  Statistics 
University  of  Missouri 
Col umbi  i,  t'O  (15201 

1  Pr  .  ,1.  Uhl  filer 

Perceptronics,  Inc. 
ft?71  V  triil  Avenue 
'ooUl and  Pills,  CR  91jrU! 

1  nr.  Howard  Uiiner 

Fure.au  of  Social  Science  Research 
1900  '1  Street,  M.  U. 

./ashing ton,  DC  200J6 

1  DR.  THOMAS  WALLSTE1I 

PSYCHOMETRIC  LABORATORY 
DAVIE  MALL  01  LA 
UNIVERSITY  OF  IIORTM  CAROL 
CHAPEL  HILL,  NC  2751 ** 

1  Dr.  David  J.  Weiss 
N(>60  F.lliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MM  55**55 

1  DR.  SUSAN  E.  UHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  (>60*14 

1  Dr.  Wolfgang  Wildgrube 
Stroitkraeft.eamt 
Rosenberg  57nO 
Bonn ,  West  Germany  D-5300 


Non  Govt 


1  Dr.  J.  Arthur  Woodward 
Department  of  Psychology 
University  of  California 
Los  Angeles,  CA  900.?*) 

1  Dr.  Robert  W'oud 

School  Examination  Department 

University  of  London 

G5-72  Gower  Street 

London  WCIE  (F.E 

ENGLAND 

1  Dr.  Karl  Zinn 

Center  for  research  on  Learning 
and  Teaching 
University  of  Michigan 
Ann  Arbor,  MI  4 510*1 


